Speech recognition has become a widely adopted and frequently used mode of interacting with computing devices. Speech input may be more convenient and efficient than traditional input modes such as typing through a keyboard. For example, mobile computing devices may offer speech recognition services as an alternative input mode to typing characters through a virtual keyboard on a touchscreen. Some computing devices are configured to accept voice commands from a user as a shortcut to performing certain actions on the computing device. Voice commands and other speech can be transcribed to text using language models. Language models have been trained using samples of text in a language to improve accuracies of the language models.